Image pickup apparatus

ABSTRACT

The image pickup apparatus includes: a first imaging sensor in which pixels including photoelectric conversion units are arranged two-dimensionally; a second imaging sensor in which pixels including photoelectric conversion units are arranged two-dimensionally, each pixel including one micro lens, and a first and a second photoelectric conversion units; a light beam splitting unit for splitting a flux of light entering an optical system into fluxes of light entering the first and the second imaging sensors separately; a first image processing unit for processing signals from the first imaging sensor, the first image processing unit generating a still image based on signals from the first imaging sensor; and a second image processing unit for processing signals from the second imaging sensor, the second image processing unit generating signals usable for focal point detection of a phase difference method and generating a moving image based on signals from the second imaging sensor.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional of application Ser. No. 14/107,649,filed Dec. 16, 2013, the entire disclosure of which is herebyincorporated by reference.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to auto-focusing and shooting a stillimage while shooting a moving image on an image pickup apparatus, andmore particularly, to an image pickup apparatus that includes multipleimaging sensors.

Description of the Related Art

There has hitherto been known a technology of performing phasedifference autofocus (AF) during the shooting of a moving image with theuse of an imaging sensor for image shooting and an AF sensor for phasedifference AF. Japanese Patent Application Laid-Open No. 2006-197406discloses a technology of performing phase difference AF whiledisplaying a moving image shot with an imaging sensor by using a halfmirror so that a subject image enters the imaging sensor and an AFsensor.

The technology disclosed in Japanese Patent Application Laid-Open No.2006-197406, however, requires stopping the moving image to shoot astill image and resuming the moving image after the shooting of thestill image is finished. In addition, using an imaging sensor thatoutputs a still image and a moving image concurrently increases theprocessing time and the circuit configuration size. The presentinvention therefore provides an image pickup apparatus capable offocusing by AF during the shooting of a moving image and shooting astill image without stopping the shooting of the moving image.

SUMMARY OF THE INVENTION

An image pickup apparatus according to one embodiment of the presentinvention has the following configuration. Specifically, the imagepickup apparatus includes: a first imaging sensor including pixels eachhaving at least one photoelectric conversion unit and arranged in atwo-dimensional array; a second imaging sensor including pixels arrangedin a two-dimensional array, each of the pixels of the second imagingsensor having one micro lens, a first photoelectric conversion unit, anda second photoelectric conversion unit; a light beam splitting unit forsplitting a flux of light entering an optical system into first andsecond fluxes of light to be applied to the first imaging sensor and thesecond imaging sensor respectively; a first image processing unit forprocessing signals from the pixels of the first imaging sensor, thefirst image processing unit generating a still image for recording basedon signals from the first imaging sensor; and a second image processingunit for processing signals from the pixels of the second imagingsensor, the second image processing unit generating, based on signalsfrom the first and second photoelectric conversion units of the secondimaging sensor, signals that are usable for focal point detection of aphase difference method, and for generating a moving image forrecording.

Further features of the present invention will become apparent from thefollowing description of embodiments with reference to the attacheddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the configuration of an image pickup apparatusaccording to a first embodiment of the present invention.

FIG. 2 illustrates the configuration of the image pickup apparatusaccording to the first embodiment of the present invention.

FIG. 3 illustrates the configuration of an imaging sensor in the firstembodiment of the present invention.

FIGS. 4A and 4B illustrate the configurations of imaging sensors in thefirst embodiment of the present invention.

FIGS. 5A and 5B illustrate the concept of focal point detection in thefirst embodiment of the present invention.

FIG. 6 is a flow chart illustrating the operation of the image pickupapparatus according to the first embodiment of the present invention.

FIGS. 7A and 7B illustrate the concept of focal point detection in thefirst embodiment of the present invention.

FIG. 8 illustrates the configuration of an image pickup apparatusaccording to a second embodiment of the present invention.

FIG. 9 illustrates the configuration of the image pickup apparatusaccording to the second embodiment of the present invention.

FIG. 10 is a flow chart illustrating the operation of the image pickupapparatus according to the second embodiment of the present invention.

FIG. 11 illustrates the operation of the image pickup apparatusaccording to the second embodiment of the present invention.

FIG. 12 illustrates the configuration of an image pickup apparatusaccording to a third embodiment of the present invention.

FIGS. 13A and 13B illustrate the configuration of imaging sensorsaccording to the third embodiment of the present invention.

FIG. 14 is a flow chart illustrating the operation of the image pickupapparatus according to the third embodiment of the present invention.

FIGS. 15A and 15B illustrate output results of the imaging sensor in thethird embodiment of the present invention.

FIG. 16 illustrates a configuration example of an image pickup apparatusaccording to a fourth embodiment of the present invention.

FIG. 17 is a schematic view illustrating the configuration of an imagepickup portion according to the fourth embodiment of the presentinvention.

FIG. 18 illustrates a configuration example of an imaging sensoraccording to the fourth embodiment of the present invention.

FIGS. 19A and 19B illustrate configuration examples of imaging sensorsaccording to the fourth embodiment of the present invention.

FIGS. 20A and 20B are explanatory diagrams of focal point detection inthe fourth embodiment of the present invention.

FIG. 21 is a flow chart illustrating the operation of the image pickupapparatus according to the fourth embodiment of the present invention.

FIG. 22 illustrates a readout area of an imaging sensor according to thefourth embodiment of the present invention.

FIGS. 23A and 23B are diagrams illustrating images that are relevant tofocal point detection in the fourth embodiment of the present invention.

FIGS. 24A and 24B are diagrams illustrating how a focal point detectionarea is selected in the fourth embodiment of the present invention.

FIGS. 25A, 25B, and 25C are diagrams illustrating readout timing of theimaging sensors according to the fourth embodiment of the presentinvention.

FIGS. 26A and 26B illustrate configuration examples of imaging sensorsaccording to a modification example of the fourth embodiment of thepresent invention.

FIGS. 27A and 27B illustrate configuration examples of imaging sensorsaccording to a fifth embodiment of the present invention.

FIG. 28 is a flow chart illustrating the operation of an image pickupapparatus according to the fifth embodiment of the present invention,and illustrates the first half of processing.

FIG. 29 is a flow chart illustrating subsequent steps of the processingof FIG. 28.

FIGS. 30A and 30B illustrate an example of shooting settings screensaccording to the fifth embodiment of the present invention.

DESCRIPTION OF THE EMBODIMENTS

In an image pickup apparatus of one embodiment of the present invention,a first imaging sensor which includes pixels and a second imaging sensorwhich includes focal point detection-use pixels are arranged so thatimages are formed by a shared image-forming optical system, and a signalfrom the pixel of the first imaging sensor and a signal from the pixelof the second imaging sensor are processed independently of each otherby image processing units. This enables the image pickup apparatus toperform an AF operation in parallel to shooting by, for example,arranging multiple imaging sensors along with a light beam splittingunit so that the imaging sensors have the same imaging surfacemagnification, shooting a still image with one of the imaging sensorsand a moving image with another of the imaging sensors independently ofeach other, and conducting focal point detection with the use of anoutput from the imaging sensor that outputs a moving image. The imagepickup apparatus is thus capable of auto-focusing during the shooting ofa moving image and shooting a still image without stopping the shootingof the moving image.

Embodiments of the present invention are described in detail below withreference to the accompanying drawings.

First Embodiment

FIG. 1 is a block diagram illustrating the configuration of an imagepickup apparatus according to a first embodiment of the presentinvention. The configuration and operation of the image pickup apparatusaccording to the first embodiment of the present invention are describedwith reference to FIG. 1. The image pickup apparatus of FIG. 1 has thefollowing configuration. A first imaging sensor 100 converts an opticalimage into electrical signals. The imaging sensor 100 is used to shootmainly a still image. An analog front end (hereinafter abbreviated asAFE) 101 performs digital conversion on an analog image signal outputfrom the imaging sensor 100 in a manner determined by gain adjustment ora predetermined quantization bit. A timing generator (hereinafterabbreviated as TG) 102 controls the driving timing of the imaging sensor100 and the AFE 101. A second imaging sensor 103 converts an opticalimage into electrical signals. The imaging sensor 103 is used to shootmainly a moving image. An AFE 104 performs digital conversion on ananalog image signal output from the imaging sensor 103 in a mannerdetermined by gain adjustment or a predetermined quantization bit. A TG105 controls the driving timing of the imaging sensor 103 and the AFE104. While this embodiment uses the AFE 101 and the TG 102 which areassociated with the first imaging sensor 100 and the AFE 104 and the TG105 which are associated with the second imaging sensor 103, aconfiguration in which an AFE and a TG are built in each imaging sensormay be employed instead.

A random access memory (RAM) 118 has a double function of an image datastoring unit, which stores image data that has been converted throughdigital conversion by the AFE 101 or the AFE 104 and image data that hasbeen processed by an image processing unit 120 or 121 described later,and a work memory, which is used when a central processing unit (CPU)124 described later operates. These functions, though implemented viathe RAM 118 in this embodiment, may be implemented via another memory aslong as the memory has a high enough access speed to cause no problems.A read-only memory (ROM) 119 stores a program that is used when the CPU124 operates. The ROM 119 in this embodiment is a flash ROM, which ismerely an example. Another memory can be employed as the ROM 119 as longas the memory has a high enough access speed to cause no problems. TheCPU 124 exerts overall control on the image pickup apparatus. The imageprocessing unit 120 performs processing such as correction andcompression on a shot still image which is described later. The imageprocessing unit 121 performs processing such as correction andcompression on a shot moving image which is described later. The imageprocessing unit 121 also has a function of adding A-image data andB-image data which are described later. In this manner, a signal from apixel of the first imaging sensor and a signal from a pixel of thesecond imaging sensor are processed independently of each other by imageprocessing units to generate images.

An AF calculation unit 122 conducts focal point detection based on apixel signal output from the second imaging sensor 103. A detachableflash memory 123 records still image data and moving image data. Therecording medium which is a flash memory in this embodiment may be otherdata writable media such as a non-volatile memory and a hard disk. Theserecording media may also be in a built-in format. An operating unit 116issues a shooting command and sets shooting conditions or otherconditions to the CPU 124. A display unit 117 displays a still image anda moving image that have been shot, a menu, and the like.

A first lens unit 111 placed at the front end of an imaging opticalsystem (shared optical system) is held in a manner that allows the firstlens unit 111 to move forward and backward in an optical axis direction.A diaphragm 110 adjusts the amount of light at the time of shooting byadjusting the diameter of its aperture. A second lens unit 109 movesforward and backward in the optical axis direction as one with thediaphragm 110, and exerts a variable magnification action (zoomfunction) in conjunction with the forward/backward movement of the firstlens unit 111. A third lens unit 108 moves forward and backward in theoptical axis direction, to thereby adjust the focal point. A half mirror107 splits an incident flux of light from a subject into reflected lightand transmitted light. Light reflected by the half mirror 107 enters thesecond imaging sensor 103 and light transmitted through the half mirror107 enters the first imaging sensor 100.

A focal plane shutter 106 adjusts the exposure time in a fraction of asecond when shooting a still image. While this embodiment uses a focalplane shutter to adjust the exposure time in a fraction of a second forthe imaging sensor 100, the present invention is not limited thereto.The imaging sensor 100 may have an electronic shutter function to adjustthe exposure time in a fraction of a second with a control pulse. Afocus driving circuit 112 is a focal point changing unit for changingthe position of the focal point of the optical system. The focus drivingcircuit 112 adjusts the focal point by driving and controlling a focusactuator 114 based on a focal point detection result of the AFcalculation unit 122, and driving the third lens unit 108 forward andbackward in the optical axis direction. A diaphragm driving circuit 113drives and controls a diaphragm actuator 115 to control the aperture ofthe diaphragm 110.

FIG. 2 is a diagram illustrating the positions of the first imagingsensor 100, the second imaging sensor 103, and the half mirror 107. Asdescribed above, the half mirror 107 is disposed at a position and anangle that cause light reflected by the half mirror 107 to enter theimaging sensor 103 and light transmitted through the half mirror 107 toenter the imaging sensor 100. In other words, light entering the firstimaging sensor 100 is one that has been transmitted through the lightbeam splitting unit and light entering the second imaging sensor 103 isone that has been reflected by the light beam splitting unit. A distancea from the center of the half mirror 107 to the imaging sensor 100 isequal to a distance b from the center of the half mirror 107 to theimaging sensor 103. Primary formed images which are subject images withan equal magnification thus enter the first imaging sensor 100 and thesecond imaging sensor 103. This configuration ensures that an imageformed on the first imaging sensor 100 is in focus even when the AFoperation is performed with the use of an image signal of the secondimaging sensor 103.

The first imaging sensor 100 is described next. FIG. 3 illustrates theconfiguration of the imaging sensor 100. The imaging sensor in FIG. 3includes a pixel array 100 a, a vertical selection circuit 100 d forselecting a row in the pixel array 100 a, and a horizontal selectioncircuit 100 c for selecting a column in the pixel array 100 a. Theimaging sensor also includes a readout circuit 100 b for reading signalsof pixels that are selected by the vertical selection circuit 100 d andthe horizontal selection circuit 100 c out of the pixels in the pixelarray 100 a. The vertical selection circuit 100 d selects a row of thepixel array 100 a and, in the selected row, activates a readout pulsewhich is output from the TG 102 based on a horizontal synchronizationsignal output from the CPU 124. The readout circuit 100 b includes anamplifier and a memory for each column, and stores pixel signals of aselected row in the memory via the amplifier. One row of pixel signalsstored in the memory are selected one by one in the column direction bythe horizontal selection circuit 100 c to be output to the outside viaan amplifier 100 e. This operation is repeated as many times as thenumber of rows until all pixel signals are output to the outside.

FIG. 4A illustrates the configuration of the pixel array 100 a. Thepixel array 100 a of the first imaging sensor 100 is made up of multiplepixels arranged in a two-dimensional array pattern in order to provide atwo-dimensional image. The pixel array 100 a of the first imaging sensor100 in FIG. 4A includes micro lenses 100 f and photodiodes (PDs) 100 gfor performing photoelectric conversion. Each pixel has one micro lens100 f for one PD 100 g, with the micro lens 100 f placed above the PD100 g. The thus configured pixels are arranged so that there are h1pixels in the horizontal direction and i1 pixels in the verticaldirection.

The configuration and reading operation of the second imaging sensor 103are the same as those of the first imaging sensor 100 illustrated inFIG. 3, and descriptions thereof are omitted. The pixel array of thesecond imaging sensor 103 is illustrated in FIG. 4B. The pixel array ofthe second imaging sensor 103 in FIG. 4B includes micro lenses 103 f,PDs 103 g, and PDs 103 h. Each pixel has one micro lens 103 f for twoPDs, with the micro lens 103 f placed above the PDs. In other words,each focal detection-use pixel has multiple photoelectric conversionunits for one micro lens. When an area where one micro lens 103 f isshared constitutes one pixel, the thus configured pixels are arranged sothat there are h2 pixels in the horizontal direction and i2 pixels inthe vertical direction. Signals accumulated in the PDs 103 g and signalsaccumulated in the PDs 103 h are separately output to the outside by thereading operation described above. A configuration described latercauses separate images having a phase difference to enter the PD 103 gand PD 103 h. Here, the PDs 103 g are therefore referred to as A-imagephotoelectric conversion units whereas the PDs 103 h are referred to asB-image photoelectric conversion units. The second imaging sensor is notlimited to the configuration of this embodiment in which two PDs areprovided for one micro lens. The second imaging sensor can employ anyconfiguration in which multiple PDs are provided for one micro lens withthe PDs placed longitudinally or transversally.

Described next are pieces of image data output by the A-imagephotoelectric conversion unit and B-image photoelectric conversion unitof the second imaging sensor 103. FIGS. 5A and 5B are diagramsillustrating the relation between a focus state and a phase differencein the imaging sensor 103. FIGS. 5A and 5B illustrate a pixel arraycross-section 103 a, the micro lenses described above, which are denotedhere by 128, the A-image photoelectric conversion units, which aredenoted here by 129, and the B-image photoelectric conversion units,which are denoted here by 130. A shooting lens 125 is an imaging opticalsystem in which an aggregation of the first lens unit 111, second lensunit 109, and third lens unit 108 of FIG. 1 is treated as one shootinglens. Light from a subject 126 passes areas of the shooting lens 125about an optical axis 127, and forms an image on the imaging sensor.Here, the centers, namely the centers of gravity, of the exit pupil andthe shooting lens coincide with each other.

With this configuration, viewing the imaging optical system from theA-image photoelectric conversion units and viewing the imaging opticalsystem from the B-image photoelectric conversion units are equivalent todividing the pupil of the imaging optical system symmetrically withrespect to the center. In other words, a flux of light from the imagingoptical system is split into two fluxes of light by what is called pupildivision. The split fluxes of light (a first flux of light and a secondflux of light) enter the A-image photoelectric conversion unit and theB-image photoelectric conversion unit which are first photoelectricconversion unit and second photoelectric conversion unit forrespectively receiving fluxes of light that have been created by pupildivision. The first flux of light is a light flux created by pupildivision in a first area of the exit pupil, and the second flux of lightis a light flux created by pupil division in a second area of the exitpupil which is off from the first area. In this manner, a flux of lightfrom a specific point on the subject 126 is split into a light flux ΦLa,which passes through a fraction of the pupil that corresponds to theA-image photoelectric conversion unit and enters the A-imagephotoelectric conversion unit, and a light flux ΦLb, which passesthrough a fraction of the pupil that corresponds to the B-imagephotoelectric conversion unit and enters the B-image photoelectricconversion unit.

These two fluxes of light, which enter from the same point on thesubject 126, pass through the same micro lens and arrive at one point onthe imaging sensor as illustrated in FIG. 5A when the imaging opticalsystem is in focus. An image signal obtained by the A-imagephotoelectric conversion unit 129 and an image signal obtained by theB-image photoelectric conversion unit 130 therefore match. When theimaging optical system is out of focus by Y as illustrated in FIG. 5B,on the other hand, a point at which the light flux ΦLa arrives and apoint at which the light flux ΦLb arrives are off from each other by anamount of change in the angle of incidence on the micro lens that isobserved in the light fluxes ΦLa and ΦLb. A phase difference isconsequently caused between an image signal obtained from the A-imagephotoelectric conversion unit 129 and an image signal obtained from theB-image photoelectric conversion unit 130. Two subject images (anA-image and a B-image) having a phase difference are respectivelyconverted through photoelectric conversion by the A-image photoelectricconversion unit 129 and the B-image photoelectric conversion unit 130,separately output to the outside, and used in an AF operation, which isdescribed later.

A pixel count h1*i1 of the first imaging sensor 100, which is forshooting a still image, is higher than a pixel count h2*i2 of the secondimaging sensor 103, which is for shooting a moving image. In otherwords, the pixel count of the second imaging sensor is lower than thepixel count of the first imaging sensor. The imaging sensor 103 which islower in pixel count than the imaging sensor 100 is larger in the planardimensions of PDs and is accordingly higher in sensitivity. A flux oflight is therefore split by the half mirror 107 so that the ratio oftransmitted light and reflected light is M:N, and the ratio N of thereflected light entering the imaging sensor 103 which is higher insensitivity is smaller than M. In other words, the intensity of a splitflux of light which enters the second imaging sensor is lower than theintensity of a split flux of light that enters the first imaging sensor.

The operation of the image pickup apparatus in this embodiment isdescribed next with reference to a flow chart of FIG. 6. First, theimage pickup apparatus stands by until a moving image shooting switchwhich is included in the operating unit 116 is pressed in a step S101.With the press of the moving image shooting switch, the shooting of amoving image is started in a step S102. When the shooting of a movingimage begins, the second imaging sensor 103, the AFE 104, and the TG 105are powered on and the CPU 124 sets moving image shooting settings.After the setting, the TG 105 outputs a readout pulse to the imagingsensor 103 based on a synchronization signal output from the CPU 124,and the imaging sensor 103 starts a reading operation at a predeterminedframe rate. This embodiment uses an electronic shutter function by wayof a slit rolling operation for the operation of accumulating andreading electric charges of a moving image, but the present invention isnot limited thereto. The imaging sensor 103 outputs A-imagephotoelectric conversion unit data and B-image photoelectric conversionunit data, which are transferred to the RAM 118 by the CPU 124, and thento the image processing unit 121. In the image processing unit 121,pieces of data of the A-image photoelectric conversion unit and theB-image photoelectric conversion unit that are below the same micro lensare added for each pixel. A frame of the moving image is created in thismanner. Thereafter, correction processing, compression, and the like areperformed on the moving image, which is then displayed on the displayunit 117 (live view). In the case where recording a moving image hasbeen selected with the use of a menu displayed on the display unit 117and the operating unit 116 prior to shooting, the moving image issequentially recorded in the flash memory 123.

In a step S103, whether or not the moving image shooting switch has beenpressed again is determined. In the case where the moving image shootingswitch has not been pressed, the shooting of the moving image iscontinued and a step S104 is executed. The shooting of the moving imageis ended when the moving image shooting switch is pressed.

In the next step which is the step S104, whether an AF switch which isincluded in the operating unit 116 has been pressed or not isdetermined. In the case where the AF switch has been pressed, AFcalculation is performed in a step S105. In the step S105, the CPU 124transfers A-image data, which is based on the A-image photoelectricconversion unit data stored in the RAM 118 and corresponds to theA-image, and B-image data, which is based on the B-image photoelectricconversion unit data stored in the RAM 118 and corresponds to theB-image, to the AF calculation unit 122.

FIG. 7A illustrates A-image data and B-image data in FIG. 5A where theoptical system is in focus. The horizontal axis represents pixelposition and the vertical axis represents output. The A-image data andthe B-image data match when the optical system is in focus. FIG. 7Billustrates A-image data and B-image data in FIG. 5B where the opticalsystem is out of focus. The A-image data and the B-image data in thiscase have a phase difference due to the situation described above, andthe pixel position of the A-image data and the pixel position of theB-image data are off from each other by a shift amount X. The AFcalculation unit 122 which is a focal point detecting unit calculatesthe shift amount X for each frame of a moving image, to therebycalculate an out-of-focus amount, i.e., the Y value in FIG. 5B. In otherwords, the focal point detecting unit performs focal point detection bya phase difference detection method with the use of an output of a focalpoint detection-use pixel of the second imaging sensor. The AFcalculation unit 122 transfers the calculated Y value to the focusdriving circuit 112.

In a step S106, the focus driving circuit 112 calculates how far thethird lens unit 108 is to be moved based on the Y value obtained fromthe AF calculation unit 122, and outputs a drive command to the focusactuator 114. The third lens unit 108 is moved by the focus actuator 114to a point where the imaging sensor 103 is in focus. Because primaryformed images having the same imaging surface magnification enter thefirst imaging sensor 100 and the second imaging sensor 103 at this pointand the depth of field and the like are the same as well, the imagingsensor 100, too, is in focus when the imaging sensor 103 is in focus.

Whether or not a still image shooting switch which is included in theoperating unit 116 has been pressed is determined next in a step S107.In the case where the still image shooting switch has been pressed, astill image is shot in a step S108. As the shooting of a still imagebegins, the first imaging sensor 100, the AFE 101, and the TG 102 arepowered on, and the CPU 124 sets still image shooting settings. Afterthe setting, the CPU 124 operates the focal plane shutter 106 to performan exposure operation on the imaging sensor 100. Thereafter, the TG 102outputs a readout pulse to the imaging sensor 100 based on asynchronization signal output from the CPU 124, and the imaging sensor100 starts a reading operation. Image data output from the imagingsensor 100 is converted into digital data by the AFE 101, and thenstored in the RAM 118. The CPU 124 transfers the image data stored inthe RAM 118 to the image processing unit 120, where correctionprocessing, compression, and the like are performed on the image data.The image data is subsequently recorded in the flash memory 123. Theprocessing then returns to the step S103 to repeat the operation of thesteps S103 to S108.

In the case where it is found in the step S104 that the AF switch hasnot been pressed, the processing moves to the step S107 to determinewhether or not the still image shooting switch has been pressed. Thesame applies to the case where the AF operation has been set to “off”via a displayed menu with the use of the display unit 117 and theoperating unit 116.

The operation described above enables the image pickup apparatus to putan image entering the imaging sensor 103 or the imaging sensor 100 intofocus by performing a phase difference AF operation while shooting amoving image (live view or moving image recording), and to shoot a stillimage at the same time.

All pixels of the second imaging sensor 103 allow for ranging and areused for phase difference AF in this embodiment. However, the secondimaging sensor is not limited to this configuration. Pixels that allowfor ranging may be arranged discretely and signals from the pixels maybe used in phase difference AF. A pixel that allows for ranging in thiscase may have one PD below a micro lens, and pupil division may beperformed with a light-shielding layer blocking light to the left orright, or the top or bottom of the PD. In other words, the secondimaging sensor only needs to have focal point detection-use pixels eachof which includes the first photoelectric conversion unit for receivinga flux of light that has been created by pupil division in the firstarea of the exit pupil, and/or the second photoelectric conversion unitfor receiving a flux of light that has been created by pupil division inthe second area of the exit pupil which is off from the first area. Theimaging sensor may include image-use pixels and focal pointdetection-use pixels.

While the second imaging sensor 103 in this embodiment includes pixelsthat allow for ranging and that are used for phase difference AF, thepresent invention is not limited to this configuration. The secondimaging sensor 103 may have the same pixel configuration as that of thefirst imaging sensor 100 in which one PD is provided below one microlens, to employ contrast AF in which an AF operation is performed bydetecting a contrast in a read moving image. In other words, the focalpoint detecting unit can detect a focal point also by a contrastdetection method in which a contrast is detected from pixel outputs ofthe second imaging sensor. Focal point detection by the contrastdetection method is of course executable also when two or more PDs areprovided below one micro lens as in this embodiment.

In addition, the present invention is not limited to the moving imagegeneration of this embodiment in which a moving image is generated byadding A-image data and B-image data of the imaging sensor 103 in theimage processing unit. In the case where each of the A-image data andthe B-image data is not necessary, for example, when focal pointdetection is not performed or is performed partially, A-image data andB-image data may be added within the imaging sensor for some of or allof the pixels before output. The present invention is also not limitedto the operation mode described in this embodiment in which a stillimage is shot during the shooting of a moving image. The image pickupapparatus is also capable of shooting a still image when a moving imageis not shot.

Second Embodiment

Now, the configuration and operation of an image pickup apparatusaccording to a second embodiment of the present invention are describedwith reference to FIG. 8. The image pickup apparatus of FIG. 8 has thefollowing configuration. A first imaging sensor 200 converts an opticalimage into electrical signals. The imaging sensor 200 is used to shootmainly a still image. An AFE 201 performs digital conversion on ananalog image signal output from the imaging sensor 200 in a mannerdetermined by gain adjustment or a predetermined quantization bit. A TG202 controls the driving timing of the imaging sensor 200 and the AFE201.

A second imaging sensor 203 converts an optical image into electricalsignals. The imaging sensor 203 is used to shoot mainly a moving image.An AFE 204 performs digital conversion on an analog image signal outputfrom the imaging sensor 203 in a manner determined by gain adjustment ora predetermined quantization bit. A TG 205 controls the driving timingof the imaging sensor 203 and the AFE 204. While this embodiment alsouses the AFE 201 and the TG 202 which are associated with the firstimaging sensor 200 and the AFE 204 and the TG 205 which are associatedwith the second imaging sensor 203, a configuration in which an AFE anda TG are built in each imaging sensor may be employed instead.

Components 206 to 224 correspond to the components 106 to 124 of thefirst embodiment, respectively. A difference is that light reflected bythe half mirror 207, which splits an incident flux of light from asubject into reflected light and transmitted light, enters the firstimaging sensor 200 whereas light transmitted through the half mirror 207enters the second imaging sensor 203.

FIG. 9 is a diagram illustrating the positions of the first imagingsensor 200, the second imaging sensor 203, and the half mirror 207. Animage transmitted through the half mirror tends to be unsharp due to anoptical aberration of the half mirror. As described later, a still imagewhich is shot with the imaging sensor 200 higher in pixel count than theimaging sensor 203 is requested to be a sharper image. The half mirror207 in this embodiment is therefore disposed at a position and an anglethat cause light reflected by the half mirror 207 to enter the imagingsensor 200 and light transmitted through the half mirror 207 to enterthe imaging sensor 203 as described above. A distance c from the centerof the half mirror 207 to the imaging sensor 200 is equal to a distanced from the center of the half mirror 207 to the imaging sensor 203.Primary formed images which are subject images with an equalmagnification thus enter the first imaging sensor 200 and the secondimaging sensor 203. This configuration ensures that an image formed onthe first imaging sensor 200 is in focus even when an AF operation isperformed with the use of an image signal of the second imaging sensor203 described later.

The configuration and function of the first imaging sensor 200 and thesecond imaging sensor 203 are the same as the configuration and functiondescribed in the first embodiment, and descriptions thereof are omitted.In the configuration of the image pickup apparatus of this embodiment,where the imaging sensor 200 is used to shoot a still image and theimaging sensor 203 is used to shoot a moving image, a pixel count h1*i1of the imaging sensor 200 is higher than a pixel count h2*i2 of theimaging sensor 203. The imaging sensor 203 which is lower in pixel countthan the imaging sensor 200 is larger in the planar dimensions of PDsand is accordingly higher in sensitivity. A flux of light is split bythe half mirror 207 so that the ratio of transmitted light and reflectedlight is M:N, and the ratio M of the transmitted light entering theimaging sensor 203 which is higher in sensitivity is smaller than N.

The operation of the image pickup apparatus in this embodiment isdescribed next with reference to a flow chart of FIG. 10. First, theimage pickup apparatus stands by until a moving image shooting switchwhich is included in the operating unit 216 is pressed in a step S201.With the press of the moving image shooting switch, the shooting of amoving image is started in a step S202. When the shooting of a movingimage begins, the imaging sensor 203, the AFE 204, and the TG 205 arepowered on and the CPU 224 sets moving image shooting settings. Afterthe setting, the TG 205 outputs a readout pulse to the imaging sensor203 based on a synchronization signal output from the CPU 224, and theimaging sensor 203 starts a reading operation at a predetermined framerate. This embodiment, too, uses an electronic shutter function by wayof a slit rolling operation for the operation of accumulating andreading electric charges of a moving image. However, the presentinvention is not limited thereto.

The imaging sensor 203 outputs A-image photoelectric conversion unitdata and B-image photoelectric conversion unit data, which aretransferred to the RAM 218 by the CPU 224, and then to the imageprocessing unit 221. In the image processing unit 221, pieces of data ofthe A-image photoelectric conversion unit and the B-image photoelectricconversion unit that are below the same micro lens are added for eachpixel. A frame of the moving image is created in this manner.Thereafter, correction processing, compression, and the like areperformed on the moving image, which is then displayed on the displayunit 217 (live view). In the case where recording a moving image hasbeen selected with the use of a menu displayed on the display unit 217and the operating unit 216 prior to shooting, the moving image issequentially recorded in the flash memory 223.

In a step S203, whether or not the moving image shooting switch has beenpressed again is determined. In the case where the moving image shootingswitch has not been pressed, the shooting of the moving image iscontinued and a step S204 is executed. The shooting of the moving imageis ended when the moving image shooting switch is pressed. In the stepS204, AF calculation is performed. The CPU 224 transfers A-image data,which is based on the A-image photoelectric conversion unit data storedin the RAM 218 and corresponds to an A-image, and B-image data, which isbased on the B-image photoelectric conversion unit data stored in theRAM 218 and corresponds to a B-image, to the AF calculation unit 222.FIG. 7A illustrates A-image data and B-image data in FIG. 5A where theoptical system is in focus. The horizontal axis represents pixelposition and the vertical axis represents output. The A-image data andthe B-image data match when the optical system is in focus. FIG. 7Billustrates A-image data and B-image data in FIG. 5B where the opticalsystem is out of focus. The A-image data and the B-image data in thiscase have a phase difference due to the situation described above, andthe pixel position of the A-image data and the pixel position of theB-image data are off from each other by a shift amount X. The AFcalculation unit 222 calculates the shift amount X for each frame of amoving image, to thereby calculate an out-of-focus amount, i.e., the Yvalue in FIG. 5B. The AF calculation unit 222 transfers the calculated Yvalue to the focus driving circuit 212. In a step S205, the focusdriving circuit 212 calculates how far the third lens unit 208 is to bemoved based on the Y value obtained from the AF calculation unit 222,and outputs a drive command to the focus actuator 214. The third lensunit 208 is moved by the focus actuator 214 to a point where the imagingsensor 203 is in focus. Because primary formed images having the sameimaging surface magnification enter the imaging sensor 200 and theimaging sensor 203 at this point and the depth of field and the like arethe same as well, the imaging sensor 200, too, is in focus when theimaging sensor 203 is in focus.

Whether or not a still image shooting switch which is included in theoperating unit 216 has been pressed is determined next in a step S206.In the case where the still image shooting switch has been pressed, theprocessing moves to a step S207. In the step S207, whether focusdriving, namely, the moving of the third lens unit 208, has stopped ornot is determined. In the case where the focus driving has not stopped,the image pickup apparatus waits until the third lens unit 208 comes toa stop. In the case where the focus driving has stopped, a still imageis shot. When the shooting of a still image begins in a step S208, theimaging sensor 200, the AFE 201, and the TG 202 are powered on, and theCPU 224 sets still image shooting settings. After the setting, the CPU224 operates the focal plane shutter 206 to perform an exposureoperation on the imaging sensor 200. Thereafter, the TG 202 outputs areadout pulse to the imaging sensor 200 based on a synchronizationsignal output from the CPU 224, and the imaging sensor 200 starts areading operation. Image data output from the imaging sensor 200 isconverted into digital data by the AFE 201, and then stored in the RAM218. The CPU 224 transfers the image data stored in the RAM 218 to theimage processing unit 220, where correction processing, compression, andthe like are performed on the image data. The image data is subsequentlyrecorded in the flash memory 223. The processing then returns to thestep S203 to repeat the operation of the steps S203 to S208.

The operation described above is outlined in FIG. 11. Between a time t1and a time t2, A-image data and B-image data are output as the operationfor one frame of a moving image. The AF calculation described above iscalculated from the A-image data and B-image data obtained between thetime t1 and the time t2, and the focus driving (the moving of the thirdlens unit 208) is executed at the time t2. The focus driving ends at atime t3. When a moving image is shot, the AF operation is performedconstantly by repeating the operation of the time t1 to the time t3. Ata time t4, the still image shooting switch is pressed but the shootingof a still image is not started because the focus driving is in progressat the time t4. After the focus driving ends at a time t5, asynchronization signal for shooting a still image is output at a time t6and the shooting of a still image begins. At a time t7, a first shuttercurtain of the focal plane shutter 206 travels first, followed by asecond shutter curtain of the focal plane shutter 206, to thereby exposethe imaging sensor 200 to light. Image data is then output from theimaging sensor 200 at a time t8, and is recorded in the flash memory 223as a still image after undergoing the processing described above. Withthis configuration in which a still image is shot after focus driving isfinished, shooting an out-of-focus image in the middle of focus drivingcan be avoided and a sharp still image can be obtained.

The operation described above enables the image pickup apparatus to putan image entering the imaging sensor 203 or the imaging sensor 200 intofocus by performing a phase difference AF operation while shooting amoving image (live view or moving image recording), and to shoot a stillimage at the same time. This embodiment, too, is receptive to themodifications described in the first embodiment.

Third Embodiment

The configuration and operation of an image pickup apparatus accordingto a third embodiment of the present invention are described below withreference to FIG. 12. The configuration of the image pickup apparatus ofthis embodiment illustrated in FIG. 12 is the same as the image pickupapparatus configuration of the first embodiment, and a descriptionthereof is omitted. This means that components 300 to 324 correspond tothe components 100 to 124 of the first embodiment, respectively. Thepositional relation of the imaging sensor 300, the imaging sensor 303,and the half mirror 307, too, is the same as that of the imaging sensorsand the half mirror in the first embodiment.

The first imaging sensor 300 of this embodiment is described. Theconfiguration of the imaging sensor 300 differs from the configurationof the imaging sensor 100 of the first embodiment in the pixel arrayportion. The pixel array portion of the imaging sensor 300 isillustrated in FIG. 13A. Illustrated in FIG. 13A are micro lenses 300 a,PDs 300 c, and PDs 300 b. Each pixel has two PDs for one micro lens,with the micro lens placed above the PDs. When an area where one microlens 300 a is shared constitutes one pixel, the thus configured pixelsare arranged so that there are j1 pixels in the horizontal direction andk1 pixels in the vertical direction. Signals accumulated in the PDs 300b and signals accumulated in the PDs 300 c are separately output to theoutside by a reading operation. Separate images having a phasedifference enter the PD 300 b and PD 300 c. Here, the PDs 300 b aretherefore referred to as A-image photoelectric conversion units whereasthe PDs 300 c are referred to as B-image photoelectric conversion units.The first imaging sensor is not limited to the configuration of thisembodiment in which two PDs are provided for one micro lens. The firstimaging sensor can employ any configuration in which multiple PDs areprovided for one micro lens with the PDs placed longitudinally ortransversally. As described above, the first imaging sensor, too, hasfocal point detection-use pixels in this embodiment.

This embodiment uses A-image data, which is based on A-imagephotoelectric conversion unit data output from the imaging sensor 300and corresponds to an A-image, and B-image data, which is based onB-image photoelectric conversion unit data output from the imagingsensor 300 and corresponds to a B-image, as follows. As described in thefirst embodiment, when the optical system is out of focus, the obtainedA-image data and B-image data have a phase difference according to theout-of-focus amount. FIGS. 15A and 15B are images obtained by shootingthe same subject. FIG. 15A shows A-image data and FIG. 15B shows B-imagedata. The data of FIG. 15A and the data of FIG. 15B are data obtainedwhen the focus is on the person whereas the background is out of focus(have a phase difference). In other words, the phase difference betweenthe A-image data and the B-image data depends on the distance from thesubject. The phase difference between the A-image data and the B-imagedata can therefore be translated into what is called parallax, and animage displayed by causing the A-image data and the B-image data toenter the left eye and the right eye separately is recognizable as astereoscopic image.

The image pickup apparatus of this embodiment has a 3D still imageshooting mode in which A-image data and B-image data are handledindependently and recorded in a format that allows the A-image data andthe B-image data to be displayed as a three-dimensional image.

The second imaging sensor 303 is described next. The configuration ofthe imaging sensor 303 is the same as that of the second imaging sensor103 of the first embodiment. A pixel array of the imaging sensor 303 isillustrated in FIG. 13B. Illustrated in FIG. 13B are micro lenses 303 a,PDs 303 b, and PDs 303 c. Each pixel has two PDs for one micro lens,with the micro lens placed above the PDs. When an area where one microlens 303 a is shared constitutes one pixel, the thus configured pixelsare arranged so that there are j2 pixels in the horizontal direction andk2 pixels in the vertical direction. Signals accumulated in the PDs 303b and signals accumulated in the PDs 303 c are separately output to theoutside by a reading operation. Separate images having a phasedifference enter the PD 303 b and PD 303 c. Here, the PDs 303 b aretherefore referred to as A-image photoelectric conversion units whereasthe PDs 303 c are referred to as B-image photoelectric conversion units.The second imaging sensor is not limited to the configuration of thisembodiment in which two PDs are provided for one micro lens.

The operation of the image pickup apparatus in this embodiment isdescribed next with reference to a flow chart of FIG. 14. The operationof steps S301 to S307 in FIG. 14 is the same as the operation of thesteps S101 to S107 described in the first embodiment, and a descriptionthereof is omitted. In a step S308, whether or not a 3D still imageshooting mode is on is determined. In the case where the 3D still imageshooting mode has been turned on prior to the shooting with the use of amenu displayed by the display unit 317 and the operating unit 316, a 3Dstill image is shot in a step S309. When the shooting of a 3D stillimage begins, the imaging sensor 300, the AFE 301, and the TG 302 arepowered on and the CPU 324 sets still image shooting settings. After thesetting, the CPU 324 operates the focal plane shutter 306 to perform anexposure operation on the imaging sensor 300. Thereafter, the TG 302outputs a readout pulse to the imaging sensor 300 based on asynchronization signal output from the CPU 324, and the imaging sensor300 starts a reading operation.

Through the reading operation, the imaging sensor 300 outputs A-imagedata and B-image data, which are converted into digital data by the AFE301, and then separately stored in the RAM 318. The CPU 324 transfersthe A-image data and B-image data stored in the RAM 318 to the imageprocessing unit 320, where correction processing, compression, and thelike are performed on the image data. The A-image data and the B-imagedata are subsequently recorded in the flash memory 323 in theirrespective predetermined formats.

In the case where it is found in the step S308 that the 3D still imageshooting mode is off, a normal still image is shot in a step S310. Whenthe shooting of a normal still image begins, the imaging sensor 300, theAFE 301, and the TG 302 are powered on and the CPU 324 sets still imageshooting settings. After the setting, the CPU 324 operates the focalplane shutter 306 to perform an exposure operation on the imaging sensor300. Thereafter, the TG 302 outputs a readout pulse to the imagingsensor 300 based on a synchronization signal output from the CPU 324,and the imaging sensor 300 starts a reading operation. Through thereading operation, the imaging sensor 300 outputs A-image data andB-image data, which are converted into digital data by the AFE 301 andthen separately stored in the RAM 318. The CPU 324 transfers the A-imagedata and B-image data stored in the RAM 318 to the image processing unit320. In the image processing unit 320, pieces of data of the A-imagephotoelectric conversion unit and the B-image photoelectric conversionunit that are below the same micro lens are added for each pixel. Anormal still image is generated in this manner. Thereafter, correctionprocessing, compression, and the like are performed on the normal stillimage, which is then recorded in the flash memory 323. The processingthen returns to a step S303 to repeat the operation of the steps S303 toS310.

The operation described above enables the image pickup apparatus to putan image entering the imaging sensor 303 or the imaging sensor 300 intofocus by performing a phase difference AF operation while shooting amoving image (live view or moving image recording), and to shoot a stillimage that can be displayed in three dimensions at the same time.

This embodiment uses the imaging sensor 303 to shoot a moving image andthe imaging sensor 300 to shoot a 3D still image, but the presentinvention is not limited to this configuration. The imaging sensor 300may be used to shoot a moving image that can be displayed in threedimensions. The image pickup apparatus may also use pixel signals fromthe imaging sensor 300 and the imaging sensor 303 both in the AFoperation. For instance, the imaging sensor 300 may have in each pixelan A-image photoelectric conversion unit and a B-image photoelectricconversion unit that are arranged transversally whereas the imagingsensor 303 has in pixel an A-image photoelectric conversion unit and aB-image photoelectric conversion unit that are arranged longitudinally,so that a phase difference in the transverse direction and a phasedifference in the longitudinal direction are detected in the imagingsensor 300 and the imaging sensor 303, respectively. In addition, thisembodiment, too, is receptive to the modifications described in thefirst embodiment.

Fourth Embodiment

FIG. 16 is a diagram illustrating a configuration example of an imagepickup apparatus 1001 according to a fourth embodiment of the presentinvention. The image pickup apparatus 1001 includes an imaging opticalsystem (image forming optical system) and multiple imaging sensors 10100and 10103. A first lens unit 10111 placed at the front end (subjectside) of the imaging optical system is supported by a lens barrel in amanner that allows the first lens unit 10111 to move forward andbackward in an optical axis direction. A diaphragm 10110 adjusts theamount of light at the time of shooting by adjusting the diameter of itsaperture. A second lens unit 10109 moves forward and backward in theoptical axis direction as one with the diaphragm 10110. The second lensunit 10109 exerts a variable magnification action (zoom function) inconjunction with the forward/backward movement of the first lens unit10111. A third lens unit 10108 is a focus lens unit which moves forwardand backward in the optical axis direction, to thereby adjust the focalpoint.

As illustrated in FIG. 17, a half mirror 10107 is a light beam splittingunit which splits an incident flux of light from a subject intoreflected light and transmitted light. Light transmitted through thehalf mirror 10107 enters the imaging sensor 10100, which is a firstimaging sensor, and light reflected by the half mirror 10107 enters theimaging sensor 10103, which is a second imaging sensor. A focal planeshutter 10106 adjusts the exposure time in a fraction of a second whenshooting a still image. While this embodiment uses the focal planeshutter 10106 to adjust the exposure time in a fraction of a second forthe first imaging sensor 10100, the present invention is not limitedthereto. The first imaging sensor 10100 may have an electronic shutterfunction to adjust the exposure time in a fraction of a second with acontrol pulse.

The first imaging sensor 10100 which converts an optical image intoelectrical signals is used to shoot mainly a still image. A first AFE10101 performs digital conversion on an analog image signal output fromthe first imaging sensor 10100 in a manner determined by gain adjustmentor a predetermined quantization bit. A first TG 10102 controls thedriving timing of the first imaging sensor 10100 and the first AFE10101.

The second imaging sensor 10103 which converts an optical image intoelectrical signals is used to shoot mainly a moving image. A second AFE10104 performs digital conversion on an analog image signal output fromthe second imaging sensor 10103 in a manner determined by gainadjustment or a predetermined quantization bit. A second TG 10105controls the driving timing of the second imaging sensor 10103 and thesecond AFE 10104. Image data output by the first AFE 10101 and imagedata output by the second AFE 10104 are transferred to a CPU 10124. Thefirst TG 10102 and the second TG 10105 generate drive signals inaccordance with control signals from the CPU 10124, and output the drivesignals to the first imaging sensor 10100 and the second imaging sensor10103, respectively. While this embodiment uses the first AFE 10101 andthe first TG 10102 which are associated with the first imaging sensor10100 and the second AFE 10104 and the second TG 10105 which areassociated with the second imaging sensor 10103, a configuration inwhich an AFE and a TG are built in each imaging sensor may be employedinstead.

The CPU 10124 exerts overall control on the image pickup apparatus. TheCPU 10124 controls a focus driving circuit 10112 and a diaphragm drivingcircuit 10113. For example, the CPU 10124 drives and controls a focusactuator 10114 via the focus driving circuit 10112 based on the resultof focal point detection (detection information) which is conducted byan AF calculation unit 10122. This causes the third lens unit 10108 tomove forward and backward in the optical axis direction as a focal pointadjusting operation. The CPU 10124 also drives and controls a diaphragmactuator 10115 via the diaphragm driving circuit 10113, therebycontrolling the aperture diameter of a diaphragm 10110.

Components 10116 to 10123 are connected to the CPU 10124. An operatingunit 10116 is operated by a user when issuing a shooting instruction andsetting shooting conditions or other conditions to the CPU 10124. Adisplay unit 10117 displays a still image and a moving image that havebeen shot, a menu, and the like. The display unit 10117 includes a thinfilm transistor (TFT) liquid crystal display, a finder, and the like atthe back of the camera main body. A RAM 10118 stores image data that hasbeen converted through digital conversion by the first AFE 10101, imagedata that has been converted through digital conversion by the secondAFE 10104, and data that has been processed by a first image processingunit 10120. The RAM 10118 further has a double function of an image datastoring unit, which stores image data that has been processed by asecond image processing unit 10121, and a work memory for the CPU 10124.These functions, though implemented via the RAM 10118 in thisembodiment, may be implemented via another memory as long as the memoryhas a high enough access speed. A ROM 119 stores a program that isinterpreted and executed by the CPU 10124. A memory device such as aflash ROM is used as the ROM 10119.

The first image processing unit 10120 performs processing such ascorrection and compression on a shot still image. The second imageprocessing unit 10121 performs processing such as correction andcompression on a shot moving image. The second image processing unit10121 also has a function of adding A-image data and B-image data whichare described later. The AF calculation unit 10122 conducts focal pointdetection based on a pixel signal output from the first imaging sensor10100. A flash memory 10123 is a detachable memory device for recordingstill image data and moving image data. The recording medium which is aflash memory in this embodiment may be other data writable media such asa non-volatile memory and a hard disk. These recording media may also bein a built-in format where the recording medium is housed in a case.

FIG. 17 is a schematic view illustrating a positional relation betweenthe first imaging sensor 10100, the second imaging sensor 10103, and thehalf mirror 10107. Light reflected by the half mirror 10107 enters thesecond imaging sensor 10103 and light transmitted through the halfmirror 10107 enters the first imaging sensor 10100. A distance e fromthe center of the half mirror 10107 to the first imaging sensor 10100 isequal to a distance f from the center of the half mirror 10107 to thesecond imaging sensor 10103 (e=f). In other words, light beams ofprimary formed images which are subject images with an equalmagnification thus enter the first imaging sensor 10100 and the secondimaging sensor 10103. This ensures that a subject image formed on thesecond imaging sensor 10103 is in focus even when AF is controlled withthe use of an image signal output by the first imaging sensor 10100.

FIG. 18 illustrates the configuration of the first imaging sensor 10100.The first imaging sensor 10100 includes a pixel array 10100 a, avertical selection circuit 10100 d for selecting a row in the pixelarray 10100 a, and a horizontal selection circuit 10100 c for selectinga column in the pixel array 10100 a. A readout circuit 10100 b readssignals of pixels that are selected by the vertical selection circuit10100 d and the horizontal selection circuit 10100 c out of the pixelsin the pixel array 10100 a.

The vertical selection circuit 10100 d selects a row of the pixel array10100 a and, in the selected row, activates a readout pulse which isoutput from the first TG 10102 based on a horizontal synchronizationsignal output from the CPU 10124. The readout circuit 10100 b includesan amplifier and a memory for each column, and stores pixel signals of aselected row in the memory via the amplifier. One row of pixel signalsstored in the memory are selected one by one in the column direction bythe horizontal selection circuit 10100 c to be output to the outside viaan output amplifier 10100 e. This operation is repeated as many times asthe number of rows until all pixel signals are output to the outside.The second imaging sensor 10103 has the same configuration, andtherefore a detailed description thereof is omitted.

FIGS. 19A and 19B each illustrate a configuration example of the pixelarray. The pixel array is made up of multiple pixel portions arranged ina two-dimensional array pattern in order to output two-dimensional imagedata. FIG. 19A is an exemplification of a pixel array configuration thatallows phase difference detection, and FIG. 19B is an exemplification ofa pixel array configuration that does not allow phase differencedetection. In this embodiment, the pixel array of the first imagingsensor 10100 has the configuration of FIG. 19A and the pixel array ofthe second imaging sensor 10103 has the configuration of FIG. 19B.

For each micro lens 10100 f, which is represented by a circular frame inFIG. 19A, multiple PDs, here, a PD 10100 g and a PD 10100 h, areprovided which are each represented by a rectangular frame. The PD 10100g and the PD 10100 h constitute multiple photoelectric conversion units.In other words, one micro lens is disposed on the subject side for everytwo PDs that constitute one pixel portion. When an area where one microlens 10100 f is shared constitutes one pixel, the thus configured pixelsare arranged so that there are 11 pixels in the horizontal direction andm1 pixels in the vertical direction. Signals accumulated in the PDs10100 g and signals accumulated in the PDs 10100 h are separately outputto the outside by a reading operation. Light beams of different imageshaving a phase difference separately enter the PDs 10100 g and the PDs10100 h as described later. Hereinafter, the PDs 10100 g are referred toas A-image photoelectric conversion units and the PDs 10100 h arereferred to as B-image photoelectric conversion units. While thisembodiment shows a configuration example in which two PDs are providedfor one micro lens, three or more PDs (for example, four PDs or ninePDs) may be provided for one micro lens. In short, the present inventionis also applicable to a configuration in which multiple PDs are providedlongitudinally or transversally for one micro lens.

In the configuration of FIG. 19B, only one PD 10103 g is provided foreach micro lens 10103 f. In other words, one micro lens is disposed onthe subject side for one PD represented by a square frame. Theconfigured pixels are arranged so that there are 12 pixels in thehorizontal direction and m2 pixels in the vertical direction. The firstimaging sensor 10100 is used to shoot a still image and the secondimaging sensor 10103 is used to shoot a moving image. The pixel count ofthe first imaging sensor 10100 (l1*m1) is therefore higher than thepixel count of the second imaging sensor 10103 (l2*m2). The secondimaging sensor 10103 which is lower in pixel count than the firstimaging sensor 10100 is larger in the planar dimensions of each PD andis accordingly higher in sensitivity. A flux of light is split by thehalf mirror 10107 so that the ratio of transmitted light and reflectedlight is M:N. The ratio is set to “N<M”, and the ratio of light enteringthe second imaging sensor 10103 which is higher in sensitivity issmaller than the ratio of light entering the first imaging sensor 10100.

Described next are image data output from the A-image photoelectricconversion units and image data output from the B-image photoelectricconversion units in the first imaging sensor 10100 which has the pixelarray configuration of FIG. 19A. FIGS. 20A and 20B are schematic viewsillustrating the relation between a focus state and a phase differencein the first imaging sensor 10100. FIGS. 20A and 20B illustrates a pixelarray cross-section 10100 a in which an A-image photoelectric conversionunit 10129 and a B-image photoelectric conversion unit 10130 areprovided for one micro lens 10128. A shooting lens 10125 is an imagingoptical system in which an aggregation of the first lens unit 10111,second lens unit 10109, and third lens unit 10108 of FIG. 16 is treatedas one lens. Light from a subject 10126 passes areas of the shootinglens 10125 about an optical axis 10127, and forms an image on theimaging sensor. Here, the centers of the exit pupil and the shootinglens coincide with each other. Light beams from different directionspass through different areas of the shooting lens 10125. With thisconfiguration, viewing the imaging optical system from the A-imagephotoelectric conversion units and viewing the imaging optical systemfrom the B-image photoelectric conversion units are therefore equivalentto dividing the pupil of the imaging optical system symmetrically. Inother words, a flux of light from the imaging optical system is splitinto two fluxes of light by what is called pupil division. The splitfluxes of light (a first light flux ΦLa and a second light flux ΦLb)respectively enter the A-image photoelectric conversion units and theB-image photoelectric conversion units.

The first flux of light from a specific point on the subject 10126 isthe light flux ΦLa, which passes through a fraction of the pupil thatcorresponds to the A-image photoelectric conversion unit 10129 andenters the A-image photoelectric conversion unit 10129. The second fluxof light from a specific point on the subject 10126 is the light fluxΦLb, which passes through a fraction of the pupil that corresponds tothe B-image photoelectric conversion unit 10130 and enters the B-imagephotoelectric conversion unit 10130. The two fluxes of light created bypupil division enter from the same point on the subject 10126 throughthe imaging optical system. The light fluxes ΦLa and ΦLb therefore passthrough the same micro lens and arrive at one point on the imagingsensor as illustrated in FIG. 20A when the subject 10126 is in focus. Animage signal obtained by the A-image photoelectric conversion unit 10129and an image signal obtained by the B-image photoelectric conversionunit 10130 accordingly have a matching phase.

When the subject is out of focus by a distance Y in the optical axisdirection as illustrated in FIG. 20B, on the other hand, a point atwhich the light flux ΦLa arrives and a point at which the light flux ΦLbarrives are off from each other by an amount of change in the angle ofincidence on the micro lens that is observed in the light fluxes ΦLa andΦLb. A phase difference is consequently caused between an image signalobtained from the A-image photoelectric conversion unit 10129 and animage signal obtained from the B-image photoelectric conversion unit10130. Two subject images (an A-image and a B-image) which are detectedby the A-image photoelectric conversion unit 10129 and the B-imagephotoelectric conversion unit 10130 and have a phase difference arerespectively converted through photoelectric conversion by the PDs. TheA-image signal and B-image signal converted by photoelectric conversionare separately output to the outside and used in an AF operation, whichis described later.

The operation of the image pickup apparatus in this embodiment isdescribed next with reference to a flow chart of FIG. 21. The followingprocessing is implemented by the CPU 10124 by reading a program out ofthe memory and executing the program. First, the image pickup apparatusstands by until a moving image shooting switch which is included in theoperating unit 10116 is pressed in a step S10101. When the moving imageshooting switch is pressed by a user's operation, the CPU 10124 startsthe shooting of a moving image in a step S10102. The second imagingsensor 10103, the second AFE 10104, and the second TG 10105 are poweredon and the CPU 10124 sets moving image shooting settings. After thesetting, the second TG 10105 outputs a readout pulse to the secondimaging sensor 10103 based on a synchronization signal output from theCPU 10124, and the second imaging sensor 10103 starts a readingoperation at a predetermined frame rate. This embodiment uses anelectronic shutter function by way of a slit rolling operation for theoperation of accumulating and reading electric charges of a movingimage. However, the present invention is not limited thereto.

The second imaging sensor 10103 outputs image data, which is transferredto the RAM 10118 by the CPU 10124. The image data is then transferred tothe second image processing unit 10121, where correction processing,compression processing, and the like are performed to create data of aframe of the moving image. The display unit 10117 displays an imagerepresented by the created data of a frame of the moving image (liveview display). In the case where the user has operated the operatingunit 10116 to choose a moving image recording mode while looking at amenu displayed on the display unit 10117 prior to shooting, the movingimage data is sequentially recorded in the flash memory 10123.

In a step S10103, the CPU 10124 determines whether or not the movingimage shooting switch has been operated again. In the case where themoving image shooting switch has not been operated, the shooting of themoving image is continued and the processing proceeds to a step S10104.The shooting of the moving image is ended when it is found in the stepS10103 that the moving image shooting switch has been operated. In thestep S10104, the CPU 10124 determines whether or not an AF switch whichis included in the operating unit 10116 has been pressed. In the casewhere it is determined that the AF switch has been pressed, theprocessing proceeds to a step S10105, where AF calculation is performedduring the shooting of the moving image. When it is found in the stepS10104 that the AF switch has not been pressed, the processing proceedsto a step S10108. In the step S10105, the CPU 10124 sets settings forreading pixel data out of phase difference detection-use pixels of thefirst imaging sensor 10100. Processing of reading pixel data out of apartial area is executed in this case, instead of reading data of theentire screen. Data is partially read out of pixel portions inside anarea 10100 i illustrated in FIG. 22. Fast AF calculation can be executedin this manner. In addition, with the operation time saved, powerconsumption is reduced. FIG. 22 illustrates reading data of three pixelsout of the area 101001 as an exemplification, but the number of pixelsto be read can be set arbitrarily. The partial area of the first imagingsensor 10100 out of which phase difference detection-use pixel data isread is changed suitably when the shooting conditions are changed or inresponse to an operation instruction or the like, and processing ofreading pixel data out of the changed area is executed. The firstimaging sensor 10100 outputs A-image photoelectric conversion unit dataand B-image photoelectric conversion unit data, which are transferred tothe RAM 10118 by the CPU 10124. The CPU 10124 transfers image data thatis based on the A-image photoelectric conversion unit data stored in theRAM 10118 (A-image data corresponding to the A-image) and image datathat is based on the B-image photoelectric conversion unit data storedin the RAM 10118 (B-image data corresponding to the B-image) to the AFcalculation unit 10122.

FIG. 23A is an exemplification of A-image data and B-image data that areobtained when the subject is in focus as illustrated in FIG. 20A. Thehorizontal axis represents pixel position and the vertical axisrepresents output level. The A-image data and the B-image data matchwhen the subject is in focus as illustrated in FIG. 23A. FIG. 23B is anexemplification of A-image data and B-image data that are obtained whenthe subject is out of focus as illustrated in FIG. 20B. The A-image dataand the B-image data in this case have a phase difference, and the pixelposition of the A-image data and the pixel position of the B-image dataare off from each other by a shift amount X. The AF calculation unit10122 calculates the shift amount X for each frame of a moving image, tothereby calculate an out-of-focus amount, i.e., the Y value in FIG. 20B.The AF calculation unit 10122 transfers the calculated Y value to thefocus driving circuit 10112 via the CPU 10124.

In a step S10106, the focus driving circuit 10112 calculates the driveamount of the third lens unit 10108 based on the Y value obtained fromthe AF calculation unit 10122, and outputs a drive command to the focusactuator 10114. The third lens unit 10108 is moved by the focus actuator10114 to an in-focus point where the subject is in focus, and the focalpoint is now located on a light receiving surface of the first imagingsensor 10100. Light beams of primary formed images having the sameimaging surface magnification enter the first imaging sensor 10100 andthe second imaging sensor 10103 at this point, and the depth of fieldand the like are the same as well. The subject is therefore in focusalso in the second imaging sensor 10103, when the subject is in focus inthe first imaging sensor 10100. In the next step which is a step S10107,the CPU 10124 determines whether or not an in-focus state has beenachieved as a result of the focus driving (in-focus determination). AFcalculation is executed again in the step S10107 to that end. Theprocessing specifics of the AF calculation are the same as in the stepS10105, and a description thereof is omitted. When it is determined thatthe imaging optical system is in an in-focus state, the processingproceeds to the step S10108. When it is determined that the imagingoptical system is not in an in-focus state, the processing returns tothe step S10104. In the step S10108, the CPU 10124 determines whether ornot a still image shooting switch which is included in the operatingunit 10116 has been pressed. In the case where the still image shootingswitch has been pressed, the processing proceeds to a step S10109. Inthe case where the still image shooting switch has not been pressed, theprocessing returns to the step S10103.

When a still image shooting operation begins in the step S10109, the CPU10124 controls the focal plane shutter 10106 to perform an exposureoperation on the first imaging sensor 10100. Thereafter, the first TG10102 outputs a readout pulse to the first imaging sensor 10100 based ona synchronization signal output from the CPU 10124. The first imagingsensor 10100 thus starts a reading operation. Image data output from thefirst imaging sensor 10100 is converted into digital data by the firstAFE 10101, and then stored in the RAM 10118. The CPU 10124 transfers theimage data stored in the RAM 10118 to the first image processing unit10120, where correction processing, compression processing, and the likeare performed on the image data. The processed image data is recorded inthe flash memory 10123. After the step S10109, the processing returns tothe step S10103 to repeat the processing steps described above. In thecase where it is found in the step S10104 that the AF switch has notbeen pressed, the processing moves to the step S10108. The same appliesto the case where the AF operation has been set to “off” through anoperation instruction issued via a displayed menu with the use of thedisplay unit 10117 and the operating unit 10116.

FIGS. 24A and 24B are diagrams exemplifying an AF frame (a frame inwhich a focal point detection area is displayed) on a shooting screen.FIG. 24A illustrates an example of AF frame selecting processing. AFframes are displayed on a screen of the display unit 10117 or the like,and the user can select an AF frame with the use of the operating unit10116. In FIG. 24A, each AF frame 10132 represents an area that can beselected, and an AF frame 10131 represents the AF frame that iscurrently selected. Processing of determining an area that correspondsto the currently selected AF frame 10131 (the area 10100 i of FIG. 22)is executed based on the position of the AF frame 10131 in the stepS10105 of FIG. 21. Pixel data of the first imaging sensor 10100 is readout of this area and the AF calculation processing is executed. Thenumber and areas of the AF frames in FIG. 24A are given as anexemplification, and can be designed arbitrarily.

FIG. 24B illustrates an example of processing of automatically selectingan AF frame by detecting the face area of the subject. The AF frames10132 and a subject image 10133 are given as an exemplification in FIG.24B. The CPU 10124 performs an image analysis on a frame of a movingimage that is being shot with the second imaging sensor 10103 (a liveview image can also be used), to thereby identify a subject and detectthe face area of the subject. An AF frame 10134 is an AF frame thatcorresponds to the face area of the subject. The AF calculationprocessing is executed based on pixel data that corresponds to theposition of the AF frame 10134. How the face of the subject is detectedis not limited to a particular method.

FIGS. 25A to 25C are diagrams illustrating readout timing of phasedifference detection-use pixels in this embodiment in comparison with anexample of a conventional case. FIG. 25A is an exemplification of timingof reading data out of phase difference detection-use pixels of a movingimage shooting-use imaging sensor in a conventional system. FIG. 25B isan exemplification of timing of reading moving image pixel data out ofthe second imaging sensor 10103 for shooting a moving image in thisembodiment. FIG. 25C is an exemplification of timing of reading phasedifference detection-use pixel data out of the first imaging sensor10100 for shooting a still image in this embodiment. Vertical axes ofFIGS. 25A to 25C represent the positions of the respective imagingsensors in the vertical direction, and horizontal axes of FIGS. 25A to25C are time axes. The length of a period 10140 of FIG. 25A indicates anaccumulation time of the imaging sensor. The accumulation time isdetermined by moving image shooting settings. In each period 10140,electric charges of signals of A-image photoelectric conversion unitsand B-image photoelectric conversion units are accumulated. Pieces ofpixel data represented by signals that are accumulated in the periods10140 are read at timing indicated by time points of oblique segments10141, and stored in the RAM by the CPU. The stored pixel data istransferred to an image processing unit, which executes processing ofadding pieces of data of the A-image photoelectric conversion unit andthe B-image photoelectric conversion unit that are below the same microlens for each pixel. Data of a frame of a moving image is created inthis manner.

The length of a period 10142 indicated by a rightwards arrow indicatesthe processing time of AF calculation. The AF calculation processing isexecuted with the use of data of A-image photoelectric conversion unitsand B-image photoelectric conversion units in the manner described withreference to FIG. 21. The length of a period 10143 indicates the focusdriving time. The focus driving processing is executed based on theresult of AF calculation in the manner described with reference to FIG.21. The system of the conventional example of FIG. 25A needs to readA-image photoelectric conversion unit data and B-image photoelectricconversion unit data in order to perform the creation of moving imageframe data and AF calculation processing at readout timing that isindicated by time points of the segments 10141. The conventional systemtherefore cannot reach its full readout throughput (in this case,processing performance par with the frame rate of the moving image). Apixel data reading method in this embodiment which solves this problemis described below with reference to FIGS. 25B and 25C.

FIG. 25B is an exemplification of processing of reading moving imagepixel data of the second imaging sensor 10103. The length of one period10140 indicates an accumulation time of the imaging sensor for shootinga moving image. Time points of oblique segments 10144 indicate pixeldata reading timing at which moving image pixel data is read. In short,only processing of reading moving image pixel data is executed at timepoints of the segments 10144, and the highest possible frame rate of themoving image can therefore be achieved.

The length of a period 10145 of FIG. 25C indicates an accumulation timeof phase difference detection-use pixels in the first imaging sensor10100. The accumulation time of phase difference detection-use pixelsdoes not need to be the same as the accumulation time of moving imagepixels (the period 10140), and can be set to an accumulation time suitedto the AF calculation processing. The accumulation time in the exampleof FIG. 25C is longer than in FIG. 25B, but the present invention is notlimited thereto. In this embodiment, where an accumulation time suitedto the AF calculation processing can be set, an AF operation of higherprecision is accomplished. Time points of segments 10146 indicate timingof reading phase difference detection-use pixel data. As has beendescribed with respect to the area 10100 i of FIG. 22, the readout time(required time) is greatly reduced by limiting data reading processingto a partial area among pixels of the first imaging sensor 10100. Afaster AF operation is accomplished as a result. The length of a period10147 indicates the processing time of AF calculation similarly to theperiod 10142 of FIG. 25A. The length of a period 10148 indicates thefocus driving time similarly to the period 10143 of FIG. 25A.

According to this embodiment, a moving image shooting operation in whichfull readout throughput is reached is realized, and AF processing inmoving image shooting is performed quickly and precisely. All pixelportions of the first imaging sensor 10100 in this embodiment includefocal point detection-use pixels so that AF processing of the phasedifference detection method can be conducted. However, the first imagingsensor is not limited to this configuration. For instance, the firstimaging sensor 10100 may include focal point detection-use pixels thatare arranged discretely and image signals that form a pair may beobtained to be used in phase difference AF processing. Each focal pointdetection-use pixel portion in this case has, for example, one PD forone micro lens, and focal point detection by a pupil division method isconducted with a light-shielding layer blocking light to the left orright, or the top or bottom, portion of the PD. Alternatively, the firstimaging sensor 10100 may have the same pixel configuration as that ofthe second imaging sensor 10103 and employ contrast AF in which an AFoperation is performed by detecting a contrast between pieces of imagedata read out of the respective pixel portions. The first imaging sensor10100 in this case only needs to have one PD for one micro lens.

The present invention is not limited to the moving image generation ofthis embodiment in which a moving image is generated by adding A-imagedata and B-image data that are obtained by the first imaging sensor10100 in the image processing unit. In the case where each of theA-image data and the B-image data is not necessary, for example, whenfocal point detection is not performed or is performed partially,A-image data and B-image data may be added within the imaging sensor forsome of or all of the pixel portions before output.

Modification Example of the Fourth Embodiment

FIGS. 26A and 26B schematically illustrate a modification example of thepixel arrays of the imaging sensors. The first imaging sensor 10100 forshooting a still image is generally configured so as to have a higherpixel count than the second imaging sensor 10103 for shooting a movingimage. On the other hand, the second imaging sensor 10103 is enhanced insensitivity by making the planar dimensions per pixel larger than thatof the first imaging sensor 10100. FIG. 26A is an exemplification of thepixel array of the first imaging sensor 10100, and FIG. 26B is anexemplification of the pixel array of the second imaging sensor 10103.One pixel (see 10103 i) of the second imaging sensor 10103 of FIG. 26Bcorresponds to four pixels (see 10100 i) in a pixel portion of the firstimaging sensor 10100 of FIG. 26A.

The ratio of the planar dimensions per pixel of the second imagingsensor 10103 and the planar dimensions per pixel of the first imagingsensor 10100 is set to approximately 4:1 for the convenience ofdescription, and the ratio can be changed to suit the specifications. Inaddition, processing of reading paired A-image photoelectric conversionunit data and B-image photoelectric conversion unit data to be used inphase difference detection can be sped up by performing additionprocessing on data of multiple pixels within the first imaging sensor10100 and then reading data obtained by the addition. For instance, whenthe ratio of the planar dimensions per pixel of the second imagingsensor 10103 and the planar dimensions per pixel of the first imagingsensor 10100 is set to N:1, data that is obtained by adding data of Npixels which corresponds to this ratio is read out of the first imagingsensor 10100. In the modification example, an out-of-focus amount can becalculated by obtaining phase difference detection signals from pixelportions within a specific area (an area selected for focal pointdetection) in the first imaging sensor 10100, which is capable of pixeloutputs of higher definition than that of the second imaging sensor10103.

Fifth Embodiment

A fifth embodiment of the present invention is described next. In thefifth embodiment, components similar to the ones in the fourthembodiment are denoted by symbols that have been used, with theexception of a first imaging sensor 10200 and a second imaging sensor10203. This is for omitting detailed descriptions of those componentsand concentrating on differences of the fifth embodiment. The basicconfiguration of an image pickup apparatus according to the fifthembodiment is the same as the configuration of FIG. 16, except for thepixel array configurations of the first imaging sensor 10200 and thesecond imaging sensor 10203. FIG. 27A illustrates a configurationexample of the pixel array of the first imaging sensor 10200, and FIG.27B illustrates a configuration example of the pixel array of the secondimaging sensor 10203.

In FIG. 27A, two PDs 10200 g and 10200 h are provided for one micro lens10200 f. When an area where one micro lens 10200 f is shared constitutesone pixel, the thus configured pixels are arranged so that there are n1pixels in the horizontal direction and of pixels in the verticaldirection. A comparison to FIG. 19A shows that pixels are distributeddensely with a higher pixel count per unit area. In FIG. 27B, two PDs10203 g and 10203 h are provided for one micro lens 10203 f. The thusconfigured pixels are arranged so that there are n2 pixels in thehorizontal direction and o2 pixels in the vertical direction. The secondimaging sensor 10203 which is used to shoot a moving image is lower inpixel count and larger in planar dimensions per pixel than the firstimaging sensor 10200 which is used to shoot a still image. The firstimaging sensor 10200 and the second imaging sensor 10203 both includephase difference detection-use pixels in this embodiment.

An imaging operation of this embodiment is described next with referenceto flow charts of FIGS. 28 and 29. The imaging operation begins in astep S10201 of FIG. 28, and the CPU 10124 determines the shooting modein a step S10202. When the shooting mode is a still image shooting mode,the processing moves to a step S10205, where a still image shootingpreparation operation is performed. When the shooting mode is a movingimage shooting mode, the processing moves from a step S10203 to a stepS10211 of FIG. 29. A user can select a shooting mode by using theoperating unit 10116, or by operating a touch panel while looking at amenu screen which is displayed on the display unit 10117.

Processing in the still image shooting mode is described first. In thestep S10205 of FIG. 28, the CPU 10124 determines the operation state ofan AF switch which is included in the operating unit 10116. Theprocessing proceeds to the step S10206 in the case where the AF switchhas been pressed. In the case where the AF switch has not been pressed,the image pickup apparatus enters a standby state and the determinationprocessing of the step S10205 is repeated. In the step S10206, AFcalculation processing is executed during the shooting of a still image.Specifically, the second imaging sensor 10203, the second AFE 10104, andthe second TG 10105 are powered on, and the CPU 10124 sets settings forreading data out of phase difference detection-use pixels of the secondimaging sensor 10203. The second imaging sensor 10203 outputs A-imagephotoelectric conversion unit data and B-image photoelectric conversionunit data, which are transferred to the RAM 10118 by the CPU 10124. TheCPU 10124 transfers A-image data, which is based on the A-imagephotoelectric conversion unit data stored in the RAM 10118 andcorresponds to an A-image, and B-image data, which is based on theB-image photoelectric conversion unit data stored in the RAM 10118 andcorresponds to a B-image, to the AF calculation unit 10122. The flowcharts of FIGS. 28 and 29 are, for the convenience of description, notfor the case where still image shooting and moving image shooting areperformed concurrently. However, in the case where live view shootinghas been specified, the CPU 10124 executes live view setting processingin the step S10206. In the case where moving image shooting is specifiedin the step S10206, the CPU 10124 executes processing of setting movingimage shooting settings.

In the next step which is a step S10207, the focus driving circuit 10112calculates the drive amount of the third lens unit 10108 based on theout-of-focus amount obtained from the AF calculation unit 10122, andoutputs a drive command to the focus actuator 10114. The third lens unit10108 is moved by the focus actuator 10114 to an in-focus point, and thefocal point is now located on a light receiving surface of the secondimaging sensor 10203. Light beams of primary formed images having thesame imaging surface magnification enter the first imaging sensor 10200and the second imaging sensor 10203 at this point, and the depth offield and the like are the same as well. Therefore, when the subject isin focus in the second imaging sensor 10203, the same subject is infocus also in the first imaging sensor 10200. In the next step which isa step S10208, the CPU 10124 determines whether or not an in-focus statehas been achieved. AF calculation is executed again to that end. Theprocessing specifics of the AF calculation are the same as in the stepS10206. When it is determined in the step S10208 that the optical systemis in an in-focus state, the processing proceeds to a step S10209. Whenit is determined that the optical system is not in an in-focus state,the processing returns to the step S10205.

In the step S10209, the CPU 10124 determines whether or not a stillimage shooting switch which is included in the operating unit 10116 hasbeen pressed. In the case where the still image shooting switch has beenpressed, the processing moves to a step S10210, where a still imageshooting operation is performed. Thereafter, the processing returns tothe S10202 through a step S10204. When it is found in the step S10209that the still image shooting switch has not been pressed, theprocessing returns to the step S10205.

The operation in the moving image shooting mode is described next withreference to FIG. 29. In the step S10211, the CPU 10124 determines theoperation state of a moving image shooting switch which is included inthe operating unit 10116. In the case where the moving image shootingswitch has been pressed, the processing moves to a step S10212, where amoving image shooting operation is started. In the case where the movingimage shooting switch has not been pressed, the processing returns tothe step S10202 of FIG. 28 via the step S10204. When the shooting of amoving image begins in the step S10212, the second imaging sensor 10203,the second AFE 10104, and the second TG 10105 are powered on and the CPU10124 sets moving image shooting settings. After the setting, the secondTG 10105 outputs a readout pulse to the second imaging sensor 10203based on a synchronization signal output from the CPU 10124. The secondimaging sensor 10203 starts a reading operation at a predetermined framerate. This embodiment uses an electronic shutter function by way of aslit rolling operation for the operation of accumulating and readingelectric charges of a moving image. The second imaging sensor 10203outputs pixel data, which is transferred to the RAM 10118 by the CPU10124. The pixel data is then transferred to the second image processingunit 10121, where correction processing, compression processing, and thelike are performed to create data of a frame of the moving image. Thedisplay unit 10117 displays (live view) a moving image represented bythe created data of a frame of the moving image on a screen. In the casewhere the user has issued an instruction to choose recording a movingimage with the use of a menu screen displayed on the display unit 10117and the operating unit 10116 prior to shooting, the moving image data issequentially recorded in the flash memory 10123.

In a step S10213, the CPU 10124 determines whether or not the movingimage shooting switch has been pressed again. In the case where themoving image shooting switch has not been pressed, the shooting of themoving image is continued and the processing proceeds to a step S10214.The shooting of the moving image is ended when the moving image shootingswitch is pressed, and the processing returns to the step S10202 of FIG.28 via the step S10204. In the step S10214, the CPU 10124 determines theoperation state of the AF switch included in the operating unit 10116.In the case where the AF switch has been pressed, the processingproceeds to a step S10215. When it is found in the step S10214 that theAF switch has not been pressed, the processing returns to the stepS10213. In the step S10215, AF calculation processing is performedduring the shooting of the moving image. Specifically, the first imagingsensor 10200, the first AFE 10101, and the first TG 10102 are powered onand the CPU 10124 sets settings for reading data out of phase differencedetection-use pixels of the first imaging sensor 10200. When readingdata out of phase difference detection-use pixels of the first imagingsensor 10200, pixel data within a limited target area is partially read,instead of reading data out of the entire area, to speed up AFcalculation. This saves the operation time and power consumption isaccordingly reduced.

In a step S10216, the focus driving circuit 10112 calculates the driveamount of the third lens unit 10108 based on the out-of-focus amountobtained from the AF calculation unit 10122, and outputs a drive commandto the focus actuator 10114. The third lens unit 10108 is moved to anin-focus point. In the next step which is a step S10217, the in-focusdetermination processing is performed after AF calculation is executedagain. The processing moves to the step S10213 when it is determinedthat the optical system is in focus. The processing returns to the stepS10214 when it is determined that the optical system is out of focus.

Through the operation described above, the imaging sensor that executesphase difference detection is switched depending on the shooting mode.Specifically, data is read out of phase difference detection-use pixelsof the second imaging sensor 10203 in the still image shooting mode, anddata is read out of phase difference detection-use pixels of the firstimaging sensor 10200 in the moving image shooting mode. By switchingimaging sensors depending on the shooting mode, the image pickupapparatus can reach its full readout throughput for moving imageshooting. In addition, the AF operation in moving image shooting isperformed quickly and precisely.

Setting operation is described next with reference to FIGS. 30A and 30B.FIGS. 30A and 30B illustrate, as examples of the display unit 10117, amoving image recording size setting screen 10230 and a subject trackingproperty setting screen 10240. In this embodiment, the operating unit10116, the display unit 10117, and the CPU 10124 perform recording sizesetting processing for setting the recording size of a moving image andframe rate setting processing for setting the frame rate of a movingimage. FIG. 30A illustrates an example of a screen for setting a movingimage recording size 10231, a moving image frame rate 10232, and amoving image compression format 10233. As the recording size of a movingimage, the user can select from, for example, the following imagequality options (shortened as “1920”, “1280”, and “640” in the drawing).

-   -   Full high vision (full high definition (HD)) image quality at        1,920 pixels by 1,080 pixels    -   High vision (HD) image quality at 1,280 pixels by 720 pixels    -   Standard image quality at 640 pixels by 480 pixels

As the frame rate of a moving image in the case where the video systemused in television broadcasting is of National Television SystemCommittee (NTSC), the user can choose from a rate of 30 frames and arate of 60 frames. In the case where the video system used in televisionbroadcasting is of Phase Alternation by Line (PAL), the user can choosefrom a rate of 25 frames and a rate of 50 frames. In an examplecinema-related use, a rate of 24 frames can be selected. Illustrated inFIG. 30A is a display example of the case where the NTSC system is used.As the moving image compression format, the user can choose from IPB inwhich data is efficiently compressed and recorded multiple frames at atime, All-I in which data is compressed and recorded one frame at atime, and the like. When the user selects an intended setting item byoperating the operating unit 10116 or other components, details of theselected setting item are displayed inside a display field 10235 on thescreen.

Depending on which moving image recording size and which moving imageframe rate are set, the second imaging sensor 10203 does not need toreach its full readout throughput in some cases. For instance, when thestandard image quality is selected as the moving image recording size,the necessary readout throughput can be kept low by thinning out piecesof pixel data of the second imaging sensor 10203. In this embodiment,when the recording size of a moving image that is set in the recordingsize setting processing is equal to or less than a threshold, the AFcalculation unit 10122 obtains pixel data output by the second imagingsensor 10203 to conduct focal point detection. The readout throughputrequired of an imaging sensor is not so high also when the moving imageframe rate is at a rate of 24 frames to 30 frames. In this embodiment,when the frame rate that is set in the frame rate setting processing isequal to or less than a threshold, the AF calculation unit 10122 obtainspixel data output by the second imaging sensor 10203 to conduct focalpoint detection. Performing phase difference detection with the secondimaging sensor 10203 reduces the overall power consumption of the systemsignificantly.

FIG. 30B is an exemplification of the subject tracking property settingscreen 10240. “Servo AF” is a function of changing the position of thefocal point of the imaging optical system in focal point detection whiletracking the subject, and is useful for the tracking of a moving objector the like. The operating unit 10116, the display unit 10117, and theCPU 10124 perform function setting processing for enabling or disablingthis function. The user uses the operating unit 10116 to operate aselecting cursor 10241 in the left-right direction, and enters a setposition with a “set” button 10243 of the operating unit 10116. The setvalue is displayed in a display field 10242 on the screen. The trackingproperty can be set with respect to a subject when the speed of a movingobject changes greatly at an instance, such as when the subject startsmoving suddenly or stops suddenly.

Depending on the settings of subject tracking property, the subjecttracking property does not need to be enhanced in some cases even whenthe image pickup apparatus is configured so that phase differencedetection is conducted with the second imaging sensor 10203. When thetracking property is not required, in other words, when the servo AFfunction is disabled (“off” in the drawing), the AF calculation unit10122 obtains pixel data output by the second imaging sensor 10203 toconduct focal point detection. Performing phase difference detectionwith the second imaging sensor 10203 reduces the overall powerconsumption of the system significantly. Quickness is not required whena subject moves at a substantially constant speed or when a trackingproperty is set with respect to a slow moving object. Therefore, whenthe set value is equal to or less than a threshold, the overall powerconsumption of the system is reduced significantly by performing phasedifference detection with the second imaging sensor 10203. When shootinga subject that makes a sudden move, rapidly accelerates or decelerates,stops abruptly, or the like, on the other hand, quickness is required ofthe tracking property of the AF operation. In the case where the setvalue exceeds a threshold, a fast and precise AF operation isaccomplished by using the second imaging sensor 10203 mainly for theshooting of a moving image and performing phase difference detectionwith the first imaging sensor 10200.

According to this embodiment, which imaging sensor is used for phasedifference detection is switched depending on shooting settings(including the recording size and the frame rate) and AF settings(including the subject tracking property). The overall power consumptionof the system can thus be reduced.

According to the present invention, an image pickup apparatus capable offocusing by AF during the shooting of a moving image and shooting astill image without stopping the shooting of the moving image can beprovided. According to the fourth and fifth embodiments of the presentinvention, a fast and precise focal point adjusting operation isaccomplished without lowering the throughput when pixel data relevant toa shot image is read.

While the present invention has been described with reference toembodiments, it is to be understood that the invention is not limited tothe disclosed embodiments. The scope of the following claims is to beaccorded the broadest interpretation so as to encompass all suchmodifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Applications No.2012-277813, filed Dec. 20, 2012, and No. 2013-111591, filed May 28,2013, which are hereby incorporated by reference herein in theirentirety.

What is claimed is:
 1. An image pickup apparatus, comprising: a firstimaging sensor comprising pixel portions arranged in a two-dimensionalarray, each of the pixel portions of the first imaging sensor comprisingone micro lens corresponding to a plurality of photoelectric conversionportions; a second imaging sensor comprising pixel portions arranged ina two-dimensional array, each of the pixel portions of the secondimaging sensor comprising one micro lens; a light beam splitting unitarranged to split a flux of light entering an optical system into firstand second fluxes of light to be led to the first imaging sensor and thesecond imaging sensor respectively; an image processing unit configuredto generate a moving image based on signals from the second imagingsensor; a calculation unit configured to, while the image processingunit generates the moving image based on signals from the second imagingsensor, perform the focus detection with a phase difference method basedon signals from the first imaging sensor.
 2. An image pickup apparatusaccording to claim 1, wherein the first imaging sensor has more pixelportions than the second imaging sensor.
 3. An image pickup apparatusaccording to claim 2, wherein the calculation unit obtains signals byadding the signals from multiple pixel portions of the first imagingsensor.
 4. An image pickup apparatus according to claim 3, wherein thecalculation unit adds signals from the multiple pixel portions of thefirst imaging sensor to produce as many number of pixel signals as thesecond imaging sensor.
 5. An image pickup apparatus according to claim1, wherein the calculation unit has a mode to perform the focusdetection of the optical system based on signals from the second imagingsensor.
 6. An image pickup apparatus according to claim 5, wherein thecalculation unit performs the focus detection based on the signals fromthe second imaging sensor while an image is taken by using the signalsfrom the first imaging sensor.
 7. An image pickup apparatus according toclaim 5, wherein each of the pixel portions of the second imaging sensorcomprises one micro lens corresponding to a plurality of photoelectricconversion portions.
 8. An image pickup apparatus according to claim 1further comprising a recording size setting unit configured to set therecording size of the moving image, wherein, when the recording size ofthe moving image set by the recording size setting unit is equal to orless than a threshold size, the calculation unit obtains signals fromthe second imaging sensor to perform the focus detection.
 9. An imagepickup apparatus according to claim 1 further comprising a frame ratesetting unit configured to set the frame rate of the moving image,wherein, when the frame rate set by the frame rate setting unit is equalto or less than a threshold rate, the calculation unit obtains signalsfrom the second imaging sensor to perform the focus detection.
 10. Animage pickup apparatus according to claim 1 further comprising afunction setting unit configured to enable a focus tracking function fora moving object, wherein, when the function is not enabled by thefunction setting unit, the calculation unit obtains signals from thesecond imaging sensor to perform the focus detection.
 11. An imagepickup apparatus according to claim 1 further comprising an selectingunit configured to select a focus detection area on a display screen,wherein the calculation unit uses signals from pixel portionscorresponding to the focus detection area.
 12. An image pickup apparatusaccording to claim 1 further comprising a face detection unit configuredto detect a face area of a subject imaged by the second imaging sensor,wherein the calculation unit uses signals from pixel portionscorresponding to the face area detected by the face detection unit. 13.An image pickup apparatus comprising: a first imaging sensor; a secondimaging sensor; a light beam splitting unit arranged to split a flux oflight entering an optical system into first and second fluxes of lightto be led to the first imaging sensor and the second imaging sensorrespectively; a focus detection unit configured to obtain signals fromthe first imaging sensor to perform the focus detection of the opticalsystem; a control unit configured to obtain a detection information fromthe focus detection unit to control the focus adjustment of the opticalsystem; an operation unit configured to instruct recording an image byusing the second imaging sensor; wherein, when the operation unit isoperated to instruct recording the image, the control unit controls thefocus adjustment of the optical system using signals from the firstimaging sensor and keeps the focus adjustment until the operation unitis operated to stop recording.
 14. An image pickup apparatus,comprising: a first imaging sensor comprising a plurality of pixelportions for the focus detection, each of which comprises one micro lenscorresponding to one photoelectric conversion portion and alight-shielding layer for shielding a part of the photoelectricconversion portion; a second imaging sensor comprising pixel portionsarranged in a two-dimensional array, signals from the pixel portions ofthe second imaging sensor used to generate an image on a display device;a light beam splitting unit arranged to split a flux of light enteringan optical system into first and second fluxes of light to be led to thefirst imaging sensor and the second imaging sensor respectively; animage processing unit configured to generate a moving image based onsignals from the second imaging sensor; a calculation unit configuredto, while the image processing unit generates the moving image based onsignals from the second imaging sensor, perform the focus detection witha phase difference method based on the signals from the first imagingsensor.